Troubleshooting the integrated Oracle DBCS (Database Cloud Service) Automatic Backup involves identifying and resolving issues related to the backup process. Here are some steps to troubleshoot common problems – Checking Backup Configuration, Storage Availability, Permissions, Connectivity, etc
To determine the root cause of job failures and whether the RMAN backup job works, the first step is to look at the RMAN log files. All RMAN log files are accessible by the user (‘oracle’) and can be found on the host under the directory: /opt/oracle/dcs/log/<SID>/rman/bkup/<oracle_unqname>
Jump to the Oracle user:
$ cd /opt/oracle/dcs/log/<SID>/rman/bkup/<oracle_unqname>
Scroll down to the related logfile of the date:
[oracle@jj2 jj2_phx153]$ pwd
/opt/oracle/dcs/log/jj2/rman/bkup/jj2_phx153
[oracle@jj2 jj2_phx153]$ ls -ltr
total 24
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:20 rman_configure
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:20 show_rman_archive_log_deletion_policy
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:20 rman_crosscheck
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:21 rman_backup
drwxr-xr-x 13 oracle oinstall 4096 Mar 12 03:22 rman_list_backup_detail
drwxr-xr-x 3 oracle oinstall 4096 Mar 12 17:32 rman_restore
[oracle@jj2 jj2_phx153]$ cd rman_backup
[oracle@jj2 rman_backup]$ ls -ltr
total 44
drwxr-xr-x 2 oracle oinstall 4096 Mar 2 03:23 2018-03-01
drwxr-xr-x 2 oracle oinstall 4096 Mar 3 03:25 2018-03-02
drwxr-xr-x 2 oracle oinstall 4096 Mar 4 03:26 2018-03-03
drwxr-xr-x 2 oracle oinstall 4096 Mar 5 03:36 2018-03-04
drwxr-xr-x 2 oracle oinstall 4096 Mar 6 03:25 2018-03-05
drwxr-xr-x 2 oracle oinstall 4096 Mar 7 03:24 2018-03-07
drwxr-xr-x 2 oracle oinstall 4096 Mar 8 03:25 2018-03-08
drwxr-xr-x 2 oracle oinstall 4096 Mar 9 03:37 2018-03-09
drwxr-xr-x 2 oracle oinstall 4096 Mar 10 03:21 2018-03-10
drwxr-xr-x 2 oracle oinstall 4096 Mar 11 03:22 2018-03-11
drwxr-xr-x 2 oracle oinstall 4096 Mar 12 03:23 2018-03-12
[oracle@jj2 rman_backup]$ cd 2018-03-12
[oracle@jj2 2018-03-12]$ ls -tlr
total 20
-rw-r--r-- 1 oracle oinstall 14401 Mar 12 03:22 rman_backup_2018-03-12_03-21-24.0169.log
-rw-r--r-- 1 oracle oinstall 1493 Mar 12 03:23 rman_backup_2018-03-12_03-23-14.0230.log
[oracle@jj2 2018-03-12]$
The above step is pure with the command line. Similarly, we can also review the job status using dbcli. We have various attributes associated with dbcli, few examples
(as opc user) $ sudo su -
(as root user) #dbcli list-databases
(as root user) #dbcli list-backupconfigs
(as root user) #dbcli list-jobs | grep -i failure
(as root user) #dbcli describe-job -i <job id> <===job id for the stuck job
Let’s explore the commands to review the outputs
[root@jj2 ~]# dbcli list-databases
ID DB Name DB Type DB Version CDB Class Shape Storage Status DbHomeID
---------------------------------------- ---------- -------- -------------------- ---------- -------- -------- ---------- ------------ ----------------------------------------
de352ede-ddf0-41a6-9627-29acc90c00c6 jj2 Si 12.1.0.2 true OLTP odb2 ASM Configured c06d7bfb-813d-4b52-9eeb-7f2ef1d3aa16
[root@jj2 ~]# dbcli list-backupconfigs
ID Name RecoveryWindow CrosscheckEnabled BackupDestination
---------------------------------------- -------------------- ---------------- ------------------- --------------------
41f6d03e-094d-45ee-aae6-ee9a1648bdd6 bgeyX3bid7o8KMG43gJH_BC 30 true ObjectStore
[root@jj2 ~]# dbcli list-jobs | grep -i failure
ID Description Created Status
---------------------------------------- --------------------------------------------------------------------------- ----------------------------------- ----------
524cb137-76ef-4806-855f-5b2ec6a74bd6 Create recovery-pitr : time '03/11/2018 13:00:00' for db : jj2 March 12, 2018 5:51:11 PM UTC Failure
4b450741-f1b8-4e0c-a3fb-8a05b4f47dab Create Longterm Backup with TAG-DBTLongterm1522439183710ORy for Db:jj2 March 30, 2018 7:51:48 PM UTC Failure
[root@jj2 ~]# dbcli describe-job -i 4b450741-f1b8-4e0c-a3fb-8a05b4f47dab
Job details
----------------------------------------------------------------
ID: 4b450741-f1b8-4e0c-a3fb-8a05b4f47dab
Description: Create Longterm Backup with TAG-DBTLongterm1522439183710ORy for Db:jj2
Status: Failure
Created: March 30, 2018 7:51:48 PM UTC
Message: DCS-10001:Internal error encountered: Wallet Status not found as OPEN-AUTOLOGIN/OPEN-UNKNOWN(UpgradeDb Case).
Task Name Start Time End Time Status
---------------------------------------- ----------------------------------- ----------------------------------- ----------
task:TaskZLockWrapper_243 March 30, 2018 7:51:51 PM UTC March 30, 2018 7:52:03 PM UTC Failure
task:TaskSequential_244 March 30, 2018 7:51:51 PM UTC March 30, 2018 7:52:03 PM UTC Failure
Database container validation March 30, 2018 7:51:51 PM UTC March 30, 2018 7:51:51 PM UTC Success
libopc existence check March 30, 2018 7:51:51 PM UTC March 30, 2018 7:51:51 PM UTC Success
Backup Validations March 30, 2018 7:51:51 PM UTC March 30, 2018 7:52:03 PM UTC Failure
In conclusion, troubleshooting the integrated Oracle DBCS Automatic Backup involves a systematic approach to identify and resolve issues affecting the backup process. Following the steps outlined above, you can effectively diagnose and address common problems encountered during backup operations.